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Asymptotic properties of Induced maximum likelihood estimators 
of non-linear models for item response variables: 
the finite-generic-item-pool case 



Douglas H. Jones 
Advanced Statistical Technologies Corporation 



Abstract 

The progress of modern mental test theory depends very much on the tech- 
niques of maximum likelihood estimation! and many popular applications make use 
of likelihoods induced by logistic item response models* While, in reality, 
item responses are nonreplicate within a single examinee and the logistic models 
are only ideal, practitioners make inferences using the asymptotic distribution 
of the maximum likelihood estimator derived as if item responses were replicated 
and satisfied their ideal model* This article proposes a sample space 
acknowledging these two realities and derives the asymptotic distribution of the 
induced maximum likelihood estimator* 

This article assumes that items, while sampled from an infinite set of 
items, have but a finite domain of alternate response functions: this situation 
is the case of the finite-generic-item-pool* Later articles will attempt to 
remove this assumption* 

Using the proposed sample space, the article applies the statistical func- 
tional approach of von Mises to derive the influence curve of the maximum like- 
lihood estimator; to discuss related robustness porperties; and to derive new 
classes of resistent estimators* This article's general purpose is revealing 
the value of these methods for uncovering the relative merits of different item 
response functions* Proofs and mathematical derivations are minimized to 
increase the accessibility of this complex subject* 
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1 . INTRODUCTION 

While maximum likelihood procedures are popular in item response theory 
(IRT), (Lord, 1980), their insensitivity to departures from assumptions is 
serious enough to warrant cautious use and further study (Wainer-Wrigh** , 1980; 
Jones, 1982). The purpose of this article is to explore the behavior of the 
procedures when the model is not true. 

To apply some of the concepts of robustness theory, we found that some of 
the more important concepts required reformulating the maximum likelihood pro- 
cedures. In particular the study of the robustness of the maximum likelihood 
estimator (MLE) requires viewing it as a function of the empirical probability 
distribution function (PDF). The original formulation of item response theory, 
as a regression problem, does not allow the summarization of the data in terms 
of an empirical PDF. In §2, we recast the structure of the problem so that the 
data can be replaced by an empirical PDF and we reformulate the MLE as a func- 
tion of it. 

In §3, we derive the asymptotic distribution of the MLE when the true PDF 
is not generated by the assumed uodel. These results are basic to understanding 
the sensitivity of the MLE to departures from assumptions. They make heavy use 
of von Mises's approach to statistical functions (Fillippova, 1982). 

In §4, we apply the asymptotic formulas derived in §3 to three popular item 
response models. A measure of goodness-of -f it , compatable with the MLE, is 
employed to play the role of the mean squared error (McCullagh-Nelder , 1983; 
Pregibon, 1981). These results reveal that certain item response models reverse 
the scale of ability. 
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In §5, we formulate the basic robustness criteria associated with Hampel's 

influence curve (IC) (Hampel, 1974; Welsch-Krasker , 1982). We derive a relation 

« 

between the IC and the maximum bias of the MLE as the true PDF is varied within 
an e -contamination neighborhood of the modeled PDF (Huber, 1981). We also 
derive the breakdown point (Huber, 1981) of the MLE for certain types of depar- 
tures from the assumptions. The analysis of these criteria shows how the notion 
of robustness in IRT is fundamentally different from linear and logistic 
regression problems. 
2. GENERAL NOTATION AND STRUCTURE 

The basic formulation of IRT based on maximum likelihood is: u=l (correct) 
or u=0 (incorrect) is observed for each item i with likelihood , given a real 
latent parameter 8, equal to 

hi(u;6) = P^e)" [l-Pi(6)]l-u 
and with Pi(6) the i th item response model. The total likelihood based on data 
ui,U2,...,u n and models ?i>?2> - — >*n is: 

n 

LOru!,.. ..Un, P!,...,P n ) = JI h^u^O). 

i=l 

For robustness studies, we need to allow for the possibility that the itea 
response models are inaccurate. Thus, we assume that E(ui)#Pi(8). But we 
retain the assumption of local independence and call P±(0) an operational model . 

To accommodate items with different difficulties and discriminating powers, 
and simultaneously, apply standard asymptotic theory, we formulate the sample 
space as: 

(Sample Space) S = {(u,x): u=0,l; xcX} 

X s finite set indexing items. 
An observation on S is denoted by s or t, etc., and is generated by administer- 
ing a randomly chosen item, x, to obtain a response, u. 
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An arbitrary probability distribution function (PDF) on S is denoted by r\ . 
A probability distribution over X is denoted by p. The conditional probability 
distribution of u given x is denoted by f(u;x). For arbitrary tj, there is a p 
and f(u;x) such that: 

H(s) = f(u;x) p(x), s = (u,x). 
Because u is binary; f(u;x) is Bernoulli with some probability of success, Il*(x) 
satisfying: 

f(u;x) = II*(x) u [l-nV)] 1 -". 
The empirical PDF defined for a sample s^,S2,..^,s n is defined by denoting 
6 s to be a point mass at s and 

*n(t) = n- 1 I « Si (t). 

i=l 1 

It is a PDF on S. The distance between two PDF 1 s £ and r\ is defined as = 

max|£(s)-n(s)| . 
seS 

A parametric family of PDF's on S is defined by {hq:8 real}. Values of hq 
are denoted by ti(s;8). A special type of a parametric family is generated by a 
set of operational models: 

Operational Models: (II(8;x) :xeX;8 real} 
Parametric Family: f (u;8,x)=n(8;x) u {l-n(8;x) }!~ u 

H(s;9)=f(u;8,x)p(x). 
The traditional structure of IRT is related as follows: the i** 1 obser- 
vation is u± with model Pi(8). Let x^ be the index value of the i t ^ 1 chosen item 
where II(8;x i )=P i (8) . Let s^u^x^, so that n(s i ;8)=f (uiiBjXiJpCxiJ^iCuije) 

P(*i). 

The likelihood based on the sample s^,S2,...,s n is: 
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n 

L(8;s 1 ,s 2 ,...,Sn) = n H(si;8). 

i=l 

I f {p( x i) : i=l,...,n} contains no information about 9, MLE's based on the two 
likelihoods are identical. 

The log-derivative of the parametric PDF is denoted by £(s;8) = (d/d8) log 
H(s;8). If it exists, a solution of the implicit equation 

n 

(Normal Equation) 0 = £ 4(Si;8) 

i-1 

is denoted by 8 n and is called the HLE. This equation simplifies with opera- 
tional models as follows: The logit of an operational model 11(8 ;x) and its 
derivative is 

g(9;x) = log n(9;x)/[l-n(9;x)] 
g'(9;x) = v(9;x) _1 IT(8;x) , where 
v(9;x) = n(9;x) [l-II(8;x)]. 
Using the definition of 1)9, we have: 

*(s;9) = g'(9;x)[u-n(9;x)] 
and the normal equation becomes 

0= l g'(9;xi) [ Ui -n(9;xi)] = £ v(9;Xi)-l [^-11(9^)] D'O^) 
i=l i=l 

The Fisher information of the parametric PDF, 119, is 

1(9) = -£ r(s;9) n(s;9) = £ *(s;9) 2 n(s;9) 

where the sum is over all s in S. This information identity follows from the 

total differential of 0 = £ 4(s;9) n(s;9); using n'(s;9) = n(s;9)(d/d9)log n(s;8) 

= Jl(s;9) 4(s;9) we have, 

0 - £ £'(s;9)n(s;9) + ^(s;9)n'(s;9) 

= I r(s;9)n(s;9) + ^(s;9) 2 n(s;9) . 

Note fcr computational purposes: 1(9) = £ g'(9;x)* v(9;x) p(x) . 
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Example. The one-parameter logistic (1PL) and two-parameter logistic (2PL) 

item response models are characterized by their logits : 

g(8;x) = a(x)[8-b(x)] where a(x) > 0, -- < b(x) < - are the 

discrimination and difficulty parameters for item x. Hence, 

n 

£(s;8) = a(x)[u-n(9;x)] and 0 = £ a(Xi)[ui - lUe^x^] is the normal 

i=l 

equation. The Fisher information is 1(0) - £ a(x) 2 v(9;x)p(x), sum 
over all X. 

We wish to generalize the normal equation in two ways: first, we want to 
show the. explicit relation between 6 n and Ti n ; second, we wish to consider esti- 
mators that are more general than MLE*s. ^ 

We rewrite the normal equation using the empirical PDF as 

0 = I Hs; 9) i n (s) 

where it will be understood that the sum is always over S. We see that the MLE 
depends explicitly on the empirical PDF, we denote this dependence by 

K = 8(in). 

If the empirical PDF is replaced by an arbitrary PDF, the normal equation 
defines a general functional relationship, 9(ti), between 9 and T) : we call 6(h) 
a statistical functional . 

We define M-type estimators generated by a score function 4>(s;9) by the 
equation 

0 = n-1 I Hs ± ; 9) = £ *(s;9)i n (s). 
i=l 

We see that *(s ;9)=£(s ;9) generates the MLE. We add this generality because our 
methods of proof in the next section are really about M-type estimators with the 
MLE results following as a special case. Note that the notion of a statistical 
function applies to M-type estimators also. More definitions that we need 
follow. 
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We define 

0(8, H) = I *(s;6)n(s) 
for an arbitrary PDF and score function. The derivative of o(8,il) with respect 
to 9 is m' (8,i|). Note that m'(8,T)0) means m'(6,ii) evaluated with 11=119 . The 
normal equation is 0=m(8,Ti n ) and Fisher's information is 1(8) = -m 1 (8,119) with 
4>=£. The Newt on-Raphe son algorithm for solving the normal equation is 

e t+i = e t + ■(et,i n )/-m l (e*,i n )j 

if tf>=£ the Fisher scoring algorithm is 

e t+l = e t + m(8 t ,ii n )/I(e t ). 
Let *(s;8) be a given score function and let 8q denote the value of 8 that 
solves the equation 0 = m(8,il), corresponding to this score function. If the 
PDF 11 is a member of some parametric family and satisfies A=il0^ for a given 
fixed parameter value 8} and if 9q=8^, then we say that the score function is 
unbiased . 

If \p is an unbiased score function, then 0=111(8,119) for all 8. This fact 
leads to an identity that is analogous to the Fisher information identity pre- 
sented previously and is proven in exactly the same way. The identity is: 

-m^Me) = I *(s;8)£(s ; 8)ii(s;e). 

If one replaces m'(8,ii n ) by its expectation under 119 in the Newt on-Raphe son 
algorithm, one obtains an algorithm that is analogous to Fisher scoring. If f 
is an unbiased score function, then one may use the above identity for -m f (8,119) 
to avoid evaluating the derivative of 4>. 

An important subclass of unbiased score functions are generated by an 
arbitrary weight function w(8;x) where 

f(s;6) = w(e;x)[u-n(e;x)]n , (e J x). 

If we choose 
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w(8;x) = v(9;x)-l 

then ♦(s;9) = 4(s;8) and we are back to the Mj£. Other choices of the weight 
function lead to resistant estimators. For example, Jones (1982) suggests 
w(8;x) = vCBjx) 11 " 1 with h£0, a tuning constant. We can stay in the class of 
exponential families with arbitrary response variable u, as long as 
II(8;x)=E(u|8,x) and wCBjxJ^arCule^)" 1 (see Jennrick and Moore, 1975). 
Jones' resistant estimator could be generated for these families also by letting 
w(8;x)= var(u|8,x) h " 1 . We obtain a more general class of estimators by allowing 
the weight functions to depend on the response: ♦(s;8)=w(8;s) [u-II(8;x) ]IT (8;x) . 
Krasker and Welsch (1982) consider these estimators for the general linear 
model. Stefanski, Carroll, and Ruppert (1984) consider these estimators for the 
logistic model . 

An algorithm based on Gauss-Newton's algorithm for solving normal equations 
with score *(s;8)=w(8;s)[u-II(8;x)]ir(8;x) is as follows: (see Holland and We]*ch, 
1977) define d\ = II'(8;Xi) and w A = w^; s^ then 

t+l t p „ t t t t t r 

8 = 8 + [I di Wi di]-l I di Wi [ui-n(8t ; xi)]. 

This algorithm is iterative reweighted least squares: at convergence 
§ n =8~, so if we define the pseudo-observation Zi=di§ n + [ui - II(§ n ;xi)] then 

6 n = [I d i w i di]" 1 I di Wi Zi . 
See Pregibon (1981) and McCullagh and Nelder (1983) for a similar algorithm 
based on the exponential family with linear predictors. Note that this 
algorithm is also identical to Fisher scoring. 



12 



8 

3. GENERAL ASYMPOTOTIC THEORY OF M-TYPE ESTIMATORS 

We present consistency and asymptotic normality (AN) results in this sec- 
tion. In the first part we confine attention to the main results and in the 
second part we supply the proofs. Readers may skip the proofs and move on to 
the next section. In the main results we discuss cc iditions for consistency and 
AN. We also characterize an approximation to the M-type estimator that is 
important for AN results and for the robustness results in §5. 
3.1 Main Results; Consisten cy 

Suppose the sample s x , s 2 ,...,s n is IID with PDF x\. The empirical PDF r\ n 
satisfies |ti n -ii|-*0 wp 1 as n+*. This fact gives us the obvious candidate for 
the limit of an M-type estimator, 6q which solves 0=m(8o,n); when does 8 n +8o? It 
is possible that the equation 0=m(8,Ti n ) yielding 6 n has more than one solution, 
iii which case a consistency result may be about only one of the possible sequen- 
ces of M-type estimators. Also, it is possible that the equation O=m(G0,n) does 
not have a local solution, in which case a consistency result would not be use- 
ful. Some known results follow. 

Huber (1964): Let 8q be the unique sotution. If *(s;8) is monotone in 8, 
for each scS then every sequence 8 n *+8Q wpl. 

Boos (1977): Let 8q be an isolated solution. If *(s,8) is continuous in 8 
for each seS then there exists a sequence 8 n +8Q wpl. 

Huber (1967, 1980): Let 8 0 be an unique solution. Let Jb(8,ti)| be bounded 
from zero as |8|+*. If tf>(s;8) is continuous in 8 for each scS, then every 
sequence 5 n +8o wpl. 

The various conditions for consistency will be satisrf i*>d when we impose 
stricter conditions for AN. 

Example. For the 2PL model 11(8 ;x) is strictly monotone increasing and hence 
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♦(s;8) = a(x)[u-II(8;x)] is monotone, the solution 6q to 0=m(8o,n)= 
I a(x)[n*(x)-n(8 0 ;x)]p(x) is unique with II*(x) arbitrary. Thus for x\ 
arbitrary, Huber (1964) applies. If T| =ri g ^ for some fixed 8^, then 
8q=8i is the unique solution. 
3.2 Main Results: Asymptotic Normality 

The first order asymptotic properties of M-type estimators are charac- 
terized by the influence curve. The influence curve (IC) is defined as: let 
seS and 6 S a point mass at s, for e>0 

IC(s,n,*) = lim e(n+€(6 s -n))-8(u) . 

€-►0 1 

Denoting 8(6 s ;e) = 8(n+€ (6 s -n)) , we see that the influence curve is an ordinary 
derivative of 8(6 S ;€) evaluated at 0: IC(s,n,*) = (d8(6 s j€)/d€)o. For M-type 
estimators, it will be proved later that for £ an arbitrary PDF, £ IC(s ,n ,4>)£(s) 
= (d8($;€)/d€)Q. This latter characterization allows us in §5 to make an impor- 
tant connection between the bias and the influence curve of MLE's. Also for M- 
type estimators the influence curve is: 
(**) IC(s,n,*) = *(s;8 0 )/-m , (eo,il) 

where throughout the remainder of this section 8q satisfies 0=m(8Q,Ti) and 
■ •(8 Uf i|)<0. 

Example. For the 2PL and H arbitrary 

IC(s,n.*) * a(x)[u-n(8 0 ;x)]/j; a(x)* v(8 0 ;x)p(x). 

Normally the denominator would depend explictly on H*(x) but does not, 

s*nce g ft (8;x)=0. However, it does depend on II*(x) through 8q which 

solves 0 = I a(x)[F*(x)-n(x;e 0 )]p(x). 

The primary application we make of the influence curve in this section is 

to get a leading term approximation : 

n 

(*) B n -e 0 * n~l I IC( Si ,n,*)+R n 

i=l 

14 
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where R n is the remainder term. We show below n£ R n «♦ 0 in probability, thus 
the behavior of ni(8 n -8o) is deduced from the approximation and the Lindeberg- 
Levy central limit theorem. The sufficient conditions for AN of M-type 
estimators are listed under (C) . They are implied by conditions (D) when the 
M-type estimator is an MLE: 

(C) There exists an open interval Qq and a constant c>0 such that for all 6 in 

*(s;8); *'(s;8); *"(s;8) exist for all seS, with the first two continuous 
in a; 

C-2: m'(8 6 ,n 6 ) < 0 for all 0 S 6 S 1 and £: |n-£|£e where 6 6 solves 0=m(8,r t ) 
and n € = u + e(£-i)) . 

(D) Define Qq = {8: v(8;x)>0 for all xeX} then suppose Qq is not empty and for 
all 8eG 0 

D-l: n'(8;x); n"(8;x); n lll (8;x) exist for all x, with the first two continuous 
in 8; 

D-2: I g ,, (8;x)[n*(x)-n(e;x)]p(x) < £ g*(8;x) 2 v(8;x)p(x) for n*(x) in an open 

interval for each x. 
Theorem 1. Assume (C) then ni(0 n -8o) is AN mean 0 and variance: 

o 0 2 = I IC(s,n,*) 2 n(s). 
Corollary. Assume (C-2) and D=D(i 1 , fixed. If i> is unbiased, i.e. 8q=6i, 

ni[8 n -6g] is AN with mean 0 and variance: 

°*o = I *(s;eo) 2 n(s;e 0 )/[X *(s;e 0 )*(s;e 0 )ii(s;eo) j 2 where t( S; e) = 

(3/36) log n(s;6). Hence an unbiased M-type estimator is efficient 
if and only if 4> is proportional to i, or in other words the MLE is 
optimal among M-type estimators with unbiased score functions. 
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3.3 Proof of Aryaptotic Normality 

Define for any £ PDF, 

Jl € = il + €(£-ii) and 
e € = 8(n € ). 

A second order Taylor series expansion about 0 is 

(***) e € = e 0 + (de/d€) 0 € + i (d 2 e/d€ 2 ) a a 2 , 

0 < a < €. Under conditions (C) we shew that this expansion is valid with €=1. 
Using the expansion with 5 = T) n , we obtain the approximation (*), where R n is 
expressed in terms of the second derivative of 8 € . To show that the IC has the 
form (**) and that ni R n + 0 in probability we derive the first two derivatives 
as follows. 

Define M(f*,€) = m(8,i| c ). Because 8 € satisfies M(8 € ,€) = 0 for all 0*€£1, 
the first and second total differentials with respect to € are identically zero 
and yield two simultaneous equations involving dJe/dcj 1=1,2: 

(1) (3M/a8)Q € d8/d€ + 3M/36 = 0 

(2) (3M/a8)e € d 2 e/d€ 2 + [(a 2 M/ae 2 )e € (de/de) + (a 2 M/aea€)e € ]de/d€ 

+ (a 2 M/3€ae)e € de/de + a 2 M/3€ 2 = o. 

Solving equations (1) and (2) for dJe/dcJ, j=l,2, and using 3M 2 /3€ 2 = C, we 
have from equation (1): 

d€ ~ ls€ )/ \3eye € 

and from equations (l') and (2): 

die _/afM\ /smV / /3mV /Vm \ /a M \ //aMV 
(2') 37* / ~ 2 \alj/\aeje £ 

Now we obtain expression (**) for the IC: Let 8q = 8(n), we have 
3M/3e = (3/3e) m(8 0 ,n € ) = (3/3e) £ *(s;8 0 ) [n(s) + e(£(s) - n(s))] 
■ I *(s;8 0 ) [C(s) - n(s)] = I *(s;8 0 ) Us). Substituting (3M/38)q 0 = m'CBo.n) 
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and the last expression into (l') we have for any PDF £, (d8/d€)o = £ *(s;8 0 ) 
£(s)/ -m f (e 0 ,n). Thus for £ = 6 S , we have IC(s,n,f) = *(s;9 0 )/ -■ , (Bq 9 h) . 

With £=Tl n , the empirical PDF, the MLE 8 n = 9(11!) where is n € with e=l. 
Using the expansion (***) with €=1 we have 

n 

e n - e 0 = n-i l ic( Si ,n,*) + R n 

i=l 

where 

R n = i^~7^ a * » f° r soine 0<a*<l. 

Now we state the conditions for the expansion (***) and hence the 
expression for R n to be valid: (see Serf ling, 1980, pp. 43, 215). 

(A) Apostle (1957, pg. 96) (d8/de) + , the righthand derivative, and 
d 2 8/de 2 exist everywhere in the open interval (0,1); with the first 
continuous in the half-closed interval [0,1). 

By expression (l') and (2') for d8/de and d 2 0/de 2 we have formulated con- 
ditions (B) that satisfy conditions (A) : 

(B) There exists an open interval Qq such that for a 11 8 in $0 
B-l: m(9,n), m^e.n), m"(8,n) exist for all n; 

B-2: there exists a constant c, such that for all £: |$-n|ic, 
n>'(0 6 ,H 6 )<O for a 11 0£e£l. 
To further obtain n^R n -»0, we need to examine the terms in expression (2') 
and place appropriate conditions on the score function, The four terms are: 

(3M/39)e 6 = m'(e 6 ,ii 6 ) 

(a 2 M/ae 2 )e 6 = m'^e^n) 

(3M/3e) = m(9 6 ,5-n) 

(3M/36ae)e 6 = m'ce, ,{-u). 
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These terms apply to R n with £=Ti n . 

We see that c. jehavior of R n depends directly on that of Ti n and in par- 
ticular the differences T) n (s) - T)(s) 9 seS. We have given condition (C) at the 
beginning of this section to keep m < (6 €> T) € ) properly away from zero so as to 
keep R n from exploding find to infer its behavior from that of ii n . The following 
two lemmas imply that n£ R n +0 wpl. 

Lean A. Assume conditions (C). Put £=n n in (2 1 ). There exists constants a 
and nQ such that |d 2 0/d€ 2 | £ a|n n -Ti| 2 for all Q£e£l and n£iiQ wpl. 
B. Assume s^,S2,...,s n are IID with PDF T) . Then as n->~ the following 
holds: 

a) Ji n (s)+ii(s) for S€S "P 1 ; 
b) (n^[Ti n (s) - T)(s)]: seS} converges in law to a Gaussian process with 

mean 0 and covariance function: 

H(s)[l-n(s)] s=t 

COV(s,t) = 



* 



-Tl(s)n(t) s#t ; 

c) |i n -i||+0 WP 1 * 

d) ni|ii n -Ti| converges in lav and in probability. 

These two lemmas imply tl:at ni R n +0 in probability and hence n£(6 n -6g) is AN 
since lemma A implies n£|R n |£a ni|T) n -T)| * |T) n -T) | and lemma B implies that |^ n ~^l^ 
while n£|i) n -ii| remains bounded in probability. 

4. SPECIFIC ASYMPTOTIC BEHAVIOR OF KLE's 

Throughout this section T)q will denote a true PDF and 6q the solution to 
0=m(6 y T]()). The asymptotic behavior of the MLE is obtained from the previous 
results of M-type estimators with score function 4(s;0), seS. We discuss these 
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aspects: goodness-of-f it , scale reproduction, and Fisher variance* 

Most practical operational models satisfy condition (D-l). Thus, we can 
say that for situations of interest, MLE's are consistent, 6 n ^6g as n-N» and is 
asymptotically normal even when i\q is not a member of the parametric family 
{ ng : 6 real} generated by the set of operational models but satisfies the mild 
regularity condition (E-2). We will employ this result with the 1PL, 2PL, and 
3PL item response models in the example at the end of this section. 

Let i)q denote a member of a certain parametric family and consider the MLE 
associated with the family* If for some 8}, t)q=iiq^, then the MLE is asymp- 
totically unbiased, meaning 6q=6^« Let denote a member of a different para- 
metric family. If 1o=€0^ then the MLE is asymptotically biased meaning Bq^B^. 
If T| o is not a member of any parametric family then the notion of unbiasedness 
has no meaning* 

Even if t\q is a member of some parametric family not identical to {119)9 
the bias does hold much information about how good the MLE may be. This is 
because the parametrization of the family containing r\$ is as good as arbitrary 
when it is not exactly the one generating the MLE. This leads us to propose a 
different notion of accuracy, possibly supplying the information we usually 
obtain with measurements of bias. 

The information supplied by the bias is obtained from comparison of its 
square to the variance, because the mean square error, a measure of total error, 
is the sum of the squared bias and variance. When the bias overwhelms the 
variance, one usually goes looking for another statistical procedure that can 
control the bias. (What this compares to in [RT is the adoption of more complex 
item response models). 
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A measure that seems to decompore into parts due to "bias" and "variance" 
and does not depend on the arbitrary parametrization of a true family of PDF's 
is as follows: for two PDF's T) and £ define 

K(Ti,$) = I T|(s) log ti(s)/$(s). 
K(ti,() is nonnegative and equal to 0 when ti=£; K'(TiQ,Tig) = -m(8,Tio), thus 
K(tiq>tiq) is minimized by 8q. A second order Taylor series expansion gives 
E K(n 0 ,Tie n ) s K(Ti 0 ,ne 0 ) - i n , (9 0 ,Tio)n" 1 aJ, 

where we have assumed that the MLE has been modified appropriately to have the 
moments for the approximation and we have used Theorem 1. 

Thus E K(no>T)8 n ) behaves as a "total error" and K(t)q,T)Qq) behaves as "bias 
squared" when it is compared to the last term on the right-hand side of the above 
Taylor series. The quantity K(T) n) T)Q n ) is proportional to the deviance in 
generalized linear models (see McCullagh and Nelder, 1983) and serves as a 
goodness-of-f it statistic. Values of E K(tiq »^ § n ) an( * its approximate 
components are displayed in Table 3 and discussed in the example at the the end 
of this section. 

Information of a different nature than bias, applicable to arbitrarily 
parametrized families, is obtained by comparing the rank-order of estimated para- 
meters with the rank-order of known abilities. Suppose there is a certain para- 
metric family {(g) of PDF's with the property that T)q € {£g} where T)q may be 
generated by any member of a population of examinees. But the family {£9} is 
too complex, making its calibration unstable with reasonable sample sizes of 
examinees. Thus, we prefer instead to use a more parsimonious family {rig] with 
the MLE obtaining 6q as a limit when T)q=£q^, 6} fix*d, and n-*». Previous 
discussion implies that Bq^Bj, in general; but the bias here is nonsense. What 



ERLC 



20 



is useful is a measure of the distortion between 6q and 8j as 0j moves 
throughout the population. We do not propost a measure but ve do think that one 
should be sensitive to reversals in the "O-scale". Table 3 displays reversals 
for the 1PL and 2PL families and is further discussed in the example at the end 
of this section. 

Turning from bias to variance, ve will now consider the predicament of 
approximating the true asymptotic variance of an MLE when ve do not know i>o* A 
ready approximation to a* of aeorem 1 in §3 is the reciprocal of Fisher's 
information: I(flo)" 1 (§ 2 )* But ve knov from §3 that it is not valid vhen i\q is 
not a member of the parametric family of PDFs that generate the MLE. 

In general, the 0* does not majorize I(8q)"^ or viceversa. Thus, it is 
possible that the reciprocal of Fisher's information can give either a conser- 
vative or misleading approximation to the true asymptotic variance. Table 3 
displays the true variance along side the Fisher variance to shov both the good 
and the bad; ve discuss this further in the example. 

Example. Listed in Table 1 are the true and modeled response probabilities of 
fiv<> subjects on four ASVAB items. The true probabilities vere actually 
obtained from a very complex item response model vhich vas calibrated or a very 
large population. The subjects are ranked from lovest to highest going left to 
right. The modeled response probabilities follov the 1PL, 2PL, or 3PL ii«s 
response models as indicated; they too vere calibrated on a very large popula- 
tion; Table 2 lists the values of the calibrated parameters. Table 3 is a sum- 
4oary of the asymptotic features of the respective MLE's, vhich ve discuss as 
follovs . 

First, note the magnitudes of the traditional notion of bias by taking 
|6]-8q| differences from columns (1) and (2). These values could easily change 
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upon reparametrization of the true response model; thus they are arbitiary. 
Thus column (1) should only convey the rank-order of the subjects. 

Our notion of "bias squared" is found in column (5), K(uo,8o). These 
values will not change if another parameterization were imposed on the true 
model. The worst fit ii 1 with subject 5(2PL), referring back to Table 1 we 

can see that the 2PL mode* provides poor estimates of all item response probabi- 
lities. There are three good values; for example, subject 5(3PL) for which 
Table 1 shows good estimates of item response probabilities. 

Column (3), -m' (8 0 ,ti 0 ) , gives us a feel for the curvature of the likelihood 
since n-m'CBQjTio) is an estimate of n*in v (6 n ,i| n ) , the second derivative of the 
log-likelihood. We see that the likelihood would tend to be flat for subjects 
1,2 (3PL) even though v '*\q,$q) shows close agreement between the estimated and 
true item response probabilities. 

We present the "total error" E K(n 0 ,H§ n ) , column (4), for a sample size of 
n=16, each item type represented equally. These errors appear to be equal 
across; models and subjects with exception of subject 5 (2PL) as noted before. 
The components of the total error are in columns (5) and (6) which can tell us 
the proportion of the total error due to systematic bias: (5)/ (4). The worst 
proportion is found with subjects 1, 2 (1PL) meaning chat the 1PL is inadequate 
with these subjects. 

We may average the "total error" and "bias squared," E K(no,T18 n ) and 
KCflO'lSg) respectively, over the subjects to get an overall assessment. These 
averages are for the 1PL, 2PL, and 3PL models respectively: (error, bias 2 ) = 
(.06,. 03), (.06,. 03), (.04,. 01). We see that on average there is at least 25 
percent of total error that is systematic bias. 
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The presence of reversals of the Og-scale can be detected from column (2) . 
Both the 1PL and 2PL item response models have reversals at the lover abilities. 
This happens because the 1PL and 2PL calibrations compensate good fit to true 
response probabilities by distorting the 9Q-scale. The Spearman rank correla- 
tion between the true ability rank-order and the 6()-scale rank-order of the 1PL, 
2PL, and 3PL models are respectively : 0.60, 0.67, 1.00. The numbers may be 
interpreted as an alternative theoretical goodness-of-f it, since we never would 
know the true rank-order of ability, there is no practical gain in the measure. 

We compare the true variance and the Fisher variance by using columns (7) 
and (8). For the most part, the Fisher variance yields a conservative 
assessment of precision; however, it can also be misleading as with subjects 3, 
5 (2PL). 

Remarks. 1) Column (3), -m'tOo,^), can play the role of information. An 

empirical assessment could be -m'(8 n ,T) n ). Also, ratios could pity the 
role of relative efficiency. 

2) One should be cautious even if measures of fit, such as K(ii n , n§ n ) , 
are favorable because as the example shows it is possible to have 
reversals of the 6(pscale even if the fit is good. 

3) We have refrained from making an elaborate comparison of the 1PL, 
2PL, and 3PL models based on the data, because one needs to properly 
account for sampling variability of the calibration process. Such a 
study is reported in Jones, Vainer and Kaplan (1984). 

5. SPECIFIC ROBUSTNESS OF THE MLE 

Let r\ denote an arbitrary true PDF, no some fixed PDF, {no} * parametric 
family of PDF 1 s that induces the MLE. Let 6(n) denote the solution to 0«m(8,n). 
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From Theorem 1 in §3 we have that 8 n +6(T|) with asymptotic variance a 2 =a 2 (ii). 
Note that we will not assume that r\ or i\q belongs tc {119} . 

The asymptotic bias of the MLE relative to i)q and {119} is defined as |6(i)) 
" 6(110) I • Let P € denote an € -neighborhood of t)q, for 11 belonging to P € we want 
to quantify the degradation of bias and variance. We say that the robustness of 
the MLE is measured by the amount of degradation of the maximum bias 

b(€) = sup |eoi) - eoi 0 )| 

T|€P € 

and the maximum variance 

v(c) = sup a*(n). 
t|cP € 

If b(e) were large relative to v(e), then the maximum variance would not be 
a very important quantifier of robustness. We confine study t* b(e) in this 
paper . 

There are several important notions for quantifying the robustness of an 
estimator. Among them are the sensitivities of a parameteric estimator, a 
fitted value, or a predicted value when one observation is deleted from the 
sample . These measures are called, respectively, gross error sensitivity 
(Huber, 1981), change in fit sensitivity and prediction sensitivity (Krasker and 
Welsch, 1983). The gross error sensitivity is related directly to the maximum 
bias as s v own below. We formulate these quantities and demonstrate their use 
with the 1PL, 2PL, and 3PL item response models. 

Another robustness notion is the sensitivity of the maximum bias as € is 
varied . Certain values of e can cause the maximum bias to explode; the smallest 
such value is called the breakdown point (Huber, 1981). We formulate this quan- 
tity and demonstrate its use with the 1PL, 2PL, and 3PL models also. 
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5.1 Sensitivities Based on Deletion 

The gross error sensitivity is defined as 

T* = max|IC(s,n 0 ,*)|. 

s 

From the leading term approximation of section 3.2, we see that it is propor- 
tional to the maximal influence exerted by anyone observation on the error of 
estimation, 8 n - 6(tiq). It is related to the maximum bias b(e) as follows. 
Recall from §3.2 that [e(n 0 +€ (£-Ho))-e(Jl())] € * IlC(s,n 0 ,*H(s) as €+0. Let P € 
be the € -contamination neighborhood defined by P € = {r\: t\ s i\q + €(£-ilo)> S 
arbitrary PDF) . Then 

sup|e(n)-8(no)| s € suplJ IC(s,n 0 ,*K(s)|. 

Thus 

b(c) s € T*. 

So that for small €, T measures the rate of growth of the maximum bias over the 
€ -contaminated neighborhood. 

For M-type estimators T = «• is equivalent to a zero breakdown point, 
meaning that any departure from tiq will cause the maximum bias to explode. 
Either condition also implies that the estimator is not continuous at iJq when 
reviewed as a function of i) (assuming, of course, a complimentary topology on 
the set of PDF's). An estimator is qualitatively robust if it is continuous 
(Huber, 1981), thus an M-type estimator is not robust if T* = • or the breakdown 
point is zero. 

The gross error sensitivity also measures the maximum change in the estima- 
tor caused by deleting one observation. Let T) n (l) and T) n (0) denote the 
empirical PDF with and without sj. Let 6 n (l) and 8 n (0) denote the corresponding 
HLE's. Then using the direct definition of the influence curve (§3.2) with 8=8^, 
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H=i) n (0) and e=l/n it is easy to show 

e n d) " 6 n (0) S n-1 IC( Si ,T| n (0),*). 

Thus 

max |9 n (l) - 9 n (0)| s n -l t* 
i 

The change in fit sensitivity concerns the effect of deleting one observa- 
tion, Si, on the estimated logit, g^jXi). This change in fit is g(6 n (l);xi) 

~ g(9 n (°); x i) s g , (9 n (°); x i)[0n( 1 ) - M 0 )]- Putting this together with the 

estimator sensitivity we have 

g(6 n (l) ;xi)-g(e n (0) ;xi)s n -lg f (9 n (0) ; Xi ) IC( Si ,^ n (0) ,*) . 

Thus the shape of g' (9;x)IC(s,n would indicate robustness as would the size 

of the change in fit sensitivity : 

Y** = M |g'(e;x)IC(s,ii,*)|. 
s 

Prediction sensitivity concerns the effect of deleting an observation from 
the sample on the predicted logit of some future item, g(8 n ;z) where z is yet to 
be administered. Let X=g f (6;z), then by a Taylor series approximation, 
g(0ni z ) s g(6 ;z)+ X(6 n -8) . Hence the change in prediction is measured by the 
change in X0 n , and XIC(s£,ti ,4>) measures this change due to deleting s±. To be 
meaningful this change must be weighed relative to its standard deviation, 
MI ICUi*!**) 2 H(s)]^« Thus the shape of the ratio indicates robustness as 
would the prediction sensitivity : 

|IC(s,ti,40| 
7 = max ~ , . 

s [I ic(s,n,*) 2 i(s)]* 

We can simplify this quantity to snow the direct dependence on the score func- 
tion by using the formula for the influence curve: 
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max 

s 



[I *(s;6)« i(s)]* 



where 6 is evaluated at 6(n). 

Now study the various sensitivities to get a feel for their implications 
• IRT using the 1PL, 2PL, and 3 r L models as examples. Graphs of these quan- 
tities are useful but require specific values for item parameters and do not 
lead to any more profound conclusions then just analytic circumspection. Graphs 
are most useful, however, with actual data, providing diagnostic information on 
the fit of the model. We study only the MLE induced by {ng} and do not look at 
general M-type estimators. We also restrict this study to sensitivities to 
departures from the parametric model, that is we let 1)o s tiq for some value of 6. 
Huber (1980) remarks that a better indication of robustness is to allow n to 
roam around a ** c neighborhood of HQ while looking at the sensitivities. We do 
not have the analytical me .ns to do tui* at this time. 



Consider now and for <.he rest of this section the MLE with operational 
models {JI(9;x):x€X}. With n 0 =T18, -Tn'(9;Tie) ■ I *(s;9) £(s;9) n(s;9) and with 
*(s;9) = £(s;9) = g' (9;x) (u-II(9;x) ] , we have 



IC(s,n,£) = 



g'(9;x)fu-II(e;x)1 
I g'(e;x) 2 v(9;x) p(x) * 



Define 



M(9 iX ) = max{7(9;x), l-II(9;x)}, 



the various sensitivies to depar^res from no = T19 are 



max g'(9;x) M(9;x) 



x 



I & f (e;x) 2 v(0;x) ,(x) 



max g'(9;x) 2 M(9;x) 



x 



and 



I g'(e;x) 2 v(9;x) p(x) 
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max g'CSjx) M(9;x) 

r= — * 

[Ig'(e;x) 2 v(9;x) p(x)]* 
Example 1. The 2PL item response models have g'(8;x) = a(x) where ->a(x)>0. if 

a(x) s ag, then the models are called the 1PL item response models. 7* is 

finite provided max a(x) is finite. If the generic item pool X is finite then 

7* is always finite. If the generic item pool is not finite, it is possible 

that sup a(x)=* but practical reasons would disallow this from happening because 

an "infinitely discriminating item" is rare. 

Example 2. The 3PL item response models are defined as II(8;x) ■ [l-c(x)] R(8;x) 
+ c(x) where 0<c(x)<l and R(9;x) is a 2 PL model. Define vj(e;x) 
= R(8;x)[l-R(9;x)]. It can be shown that g'(8;x) = [l-c(x)]a(x) vi(B;x)/v(B;x) . 
T is finite provided max a(x) is finite, the discussion in the previous example 
applies here too. 

Example 3. For all the 1PL, 2PL, and 3 PL models, because of the behavior of 
g'(8;x), T** and T are finite if and only if max a(x) is finite. T** and T, but 
not T , are invariant for changes of scale in a(x) and b(x). Presumably c(x) is 
scale free as it is a probability of the examinee guessing the correct answer to 
item x. 

The examples lead to the general conclusion that T, T* f and T** are finite 
if and only if maxig'cejx)! is finite. For the 1PL, 2PL, and *PL models this 
condition is equivalent to having max a(x) finite. 

Because the sensitivities change as 6 changes, their variation over the 
entire range of practical 6-values should be studied to properly assess robust- 
ness in IRT. This allows for the tact that the MLE procedure must estimate 
unique 9 parameters for different subjects. This is in marked contrast with 
estimation in logistic regression — the same estimation procedures as IRT but 
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the object is to estimate a single 8 (such as lethal dose 50 or the vector of 
parameters in one response function) . Because the sensitivities must be viewed 
globally, procedures that are robust for logistic regression may not be directly 
transferable to IRT. 

Consider what happens as |8|-*«. The denominator of 7* and 7** is Fisher's 
information; for 7, it is just the square root. For extreme 8's it is reason- 
able to assume that any finite set of generic items X, item responses hold 
little information about 8; thus it is probable that the denominators of the 
sensitivities approach zero as |6|+~. Unless the numerators approach zero at 
the same or faster rate as the denominators, the sensitivities will explode. 
Applying this idea to each sensitivity, we conclude that 7* always explodes and 
for models with v(8;x)+0, 7* and 7** both explode. Of the models considered 
before, the 3PL is the only one having v(8;x)>0 as 8+--; thus 7* and 7** are 
bounded for the negative extremes of ability. 

These results imply that the MLE procedures are not robust because the 
maximum b. in an e-contaminated neighborhood is approximately €7* and 7* is 
unbounded as |8|-n»; thus, the MLE cannot tolerate any contamination at extreme 
8. The .'PL fairs a little better than the 1PL or 2PL as 8+-« since its gross 
error sensitivity grows a little slower. Thus to achieve full protection one 
must look outside the class of MLE procedures, which means we have to sacrifice 
efficiency. (Contrast this with the location problem There the median is the 
efficient procedure for logistic errors and jt is optimal for minimizing the 
maximum bias; Huber, 1981). 
5.2 Breakdown Point 

The worst possible bias at r\ 0 is defined as b(l) * sup| 8(£)-8(Ho) I ■ where 
the supremum is over all arbitrary PDFs, £. Let P € be an € -neighborhood of no- 
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The breakdown point , €*, is the largest 6 for which b(:) is less than the worst 
value : 

e* = sup{e : b(e) < b(l)}. 
The value of € depends on the kind of P € chosen; however, it is sometimes 
adequate to consider just one kind of neighborhood. In IRT, bO)* 5 *. 

We use the following kind of € -neighborhood: Let and define 

v = II(l-II). Denote the interval D(8;x) = [II - cv*, II + cvi] when II = II(8,x); 
8 is fixed. Denote the subinterval of [0,l] by D*(8;x) = DCBjx^Ojl] . The 
collection of intervals (D*(8;x)}, x fixed, is an €-envelope of the item 
response function P(8;x). Define P^q = {n : n(s) = H*(x) u (l-lT^x)] 1 - 11 p(x); 
J!*(x)€ D*(8;x)}. It is an € -neighborhood "centered" at t\q. 

Define b + (c) = sup[8($)-8(n 0 ) ] and b-( € ) = inf [8($)-8(Ho)] . Then 
b(€) = max{b + (€), - b_(c)}. We consider b + (c) first. 

Let tiq = tiq 0 . Define n + (8 0 ;x) = min{l ,II+€v£} with n=n(8 0 ;x). It is clear 
that n + (8 0 ;x) € D(8q;x) and hq q . the corresponding PDF, satisfies m(8;nQ 0 )>m(8;n) 
for all 8 and all r\ € P€,8q- Th® maximum "positive" bias satisfies 

b + (€) = inf{8 : m(8;ne 0 ) < 0} - 8 0 . 

We have breakdown if b+(€) = b(l) = ••. To avoid this it is necessary that 

€ satisfy lim m(8;ng ) < 0. Using the definition of m(8;n) we have 
8-m. u 

«(Me 0 ) = Z 8 , ( 9 ;x)[n(8 0 ;x) - n(8;x)]p(x) + € I g^ejxMBoJxjipCx). 
Letting 8-*t and denoting g'(«»;x) = lim g'(8;x) we have an equation for the 
"positive" side breakdown: 

€+ = I g'(-;x) [l-n(8 0 ;x)] p(x) . 
I g'(-;x) v(8 0 ;x)4 p(x) 

Similarly the "negative" side breakdown is: 
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e - = I g'(">;x) n(6 0 ;x) p(x) . 
I g'(-;x) v(6 0 ,x)i p(x) 

And the breakdown: 

e + = min(e + , e~). 

For a fixed Bq, all MLE procedures for IRT have a breakdown point that is 
not 0 for P € neighborhoods considered thus far. But as |8q|-»« the picture 
changes: €* = 0 if either n(8 0 ;x)-»l or 0 for all items x. Thus the 1PL and 
2PL induced MLEs have zero breakdown, meaning they have no tolerance for depar- 
tures from their models. The 3 PL induced MLE has zero breakdown, but for 8-^-», 
the "negative" sided breakdown is not zero, so it could tolerate some departure 
from its model there. 

Example. The following displays the "positive" and "negative" breakdown points 
for the 3PL model with a(x) s ag and c(x) s cq. 

co e~ e + 

.025 .16 0 

.05 .23 0 

.10 .33 0 

.20 .50 0 
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TABLE 1. ITEM RESPONSE PROBABILITIES 



MODEL 


ITEM 1 


ITEM 2 


ITEM 3 


ITEM 4 


Subject:! 


True 


.42 


.27 


.03 


.36 


1PL 


.32 


.30 


.19 


.32 


2 PL 


.30 


.30 


.08 


.30 


3PL 


.26 


.25 


.04 


.23 


Subjects 


True 


.32 


.26 


.05 


.26 


1PL 


.26 


.25 


.15 


.10 


2PL 


.27 


.27 


.05 


.26 


3PL 


.26 


.26 ' 


.05 


.23 


Subject: 3 


True 


.19 


.28 


.11 


.20 


1PL 


.22 


.21 


.12 


.07 


2 PL 


.27 


.27 


.05 


.26 


2PL 


.27 


.28 


.10 


.24 


Subject: 4 


True 


.50 


.44 


.47 


.60 


1PL 


.55 


.53 


.38 


,55 


2PL 


.46 


.46 


.50 


.53 


3PL 


.46 


.43 


.46 


.41 


Subject: 5 


True 


.89 


.77 


.88 


.95 


1PL 


.88 


.88 


.79 


.88 


2PL 


.72 


.72 


.98 


.84 


3PL 


.91 


.73 


.85 


.96 
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TABLE 2. PARAMETERS OF OPERATIONAL MODELS [t\q] 



1-PL 



I tea 


b 


a 


c 


1 


.9 


.7 


0 


2 


1.0 


.7 


0 


3 


1.9 


.7 


0 


4 


.9 


.7 


0 



2-PL 



1 


1.4 


.5 


0 


2 


1.4 


.5 


0 


3 


1.1 


1.7 


0 


4 


.9 


.7 


0 



3-PL 



1 


1.3 


3.2 


.26 


2 


1.6 


1.9 


.25 


3 


1.1 


2.1 


.03 


4 


1.1 


3.5 


.23 
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TABLE 3. ASYMPTOTIC PARAMETERS OF MLE's 





(1) 


(2) t 


(3) 


(4)* 


CO 


(6)* 


(7) 


(8) 




Model 


«1 


e 0 -v 


(8o»"o) 


EK(io»ie 0 ) 




-^■"(BC^O) 11 " 


.1 2 2 
L <*0 ff 0 


Wo)" 1 




Subject:! ' 


1PL 


-2 


-.2(3) 


.10 


.07 


.04 


.03 


9.47 


10.26 




2PL 


-2 


-.3(3) 


.10 


.04 


.02 


.02 


7.61 


9.76 




3PL 


-2 


-1.2(1) 


.00+ 


.03 


.03 


.00+ 


400.00 


400.00 




Subject: 2 




















1PL 


-1 


-.6(2) 


.10 


.07 


.04 


.03 


8,68 


10.26 




2PL 


-1 


-.6(1.5) 


.09 


.03 


.00+ 


.03 


11.42 


11.76 


' i 


3PL 


-1 


-.8(2) 


.01 


.04 


.00+ 


.04 


133.33 


133.33 




Subject: 3 


















i 


1PL 


0 


-.9(1) 


.08 


.06 


.03 


.03 


13.33 


13.33 




2PL 


0 


-.6(1.5) 


.09 


.06 


.02 


.04 


15.92 


11.76 




3PL 


0 


-.1(3) 


.06 


.04 


.01 


.03 


18.06 


16.67 




Subject: 4 




















1PL 


1 


1.2(4) 


.12 


.04 


.01 


.03 


8.16 


8.16 




2PL 


1 


1.1(4) 


.24 


.05 


.02 


.03 


4.12 


4.12 




3PL 


1 


1-0(4) 


.96 


.05 


.02 


.03 


1.03 


1.04 




5 

Subject: 5 


1PL 


2 


3.8(5) 


.06 


.06 


.03 


.03 


14.08 


16.00 




2PL 


2 


3.3(5) 


.06 


.14 


.07 


.07 


36.36 


18.18 




3PL 


2 


1.9(5) 


.60 


.03 


.00+ 


.03 


1.68 


1.66 
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^ Rank order Appears In parenthesis. 
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